Introduction to Machine Learning

Unit 09: PCA Contd, Experiments & Naive Bayes

1. Introduction

This unit continues our exploration of Principal Component Analysis (PCA). We deep-dive into explained variance ratios, work through a complete hand-computed PCA example on the Iris dataset, then run a comparative case study pitting filter methods, wrapper methods, and PCA against each other on the Adult Census Income dataset. Finally, we introduce the mathematical foundation of the Naive Bayes classifier starting from Bayes' Theorem.

            Learning Objectives
            Compute and interpret Explained Variance Ratios from eigenvalues and select \(k\) components via a scree plot
Execute a step-by-step hand calculation of PCA on a small dataset (Iris)
Compare Filter, Wrapper, and PCA feature engineering experimentally via AUC on Adult Census data
State and explain Bayes' Theorem and the conditional independence "naive" assumption
Derive the Naive Bayes classification rule from first principles

        

2. Theory

2.1 PCA Recap — Step-by-Step Process

PCA has two distinct phases. The parameters (mean, std, projection matrix $W$) are learned on the training set only.

Learning Phase

Application Phase

Standardize the $d$-dimensional training dataset ($X_{\text{train}}$)
Construct the covariance matrix $\Sigma$ of shape $d \times d$
Decompose $\Sigma \rightarrow$ eigenvectors + eigenvalues via eigendecomposition
Sort eigenvalues in decreasing order to rank eigenvectors

Select top $k$ eigenvectors corresponding to the $k$ largest eigenvalues ($k \lt d$)
Build projection matrix $W$ of shape $d \times k$ from top-$k$ eigenvectors
Transform both $X_{\text{train}}$ and $X_{\text{test}}$ using $W$ to get $k$-dimensional features

2.2 Explained Variance Ratio & Scree Plot

Each eigenvalue $\lambda_j$ quantifies the variance captured by its corresponding principal component. The proportion of total variance explained by component $j$ is:

\[ \text{EVR}_j = \frac{\lambda_j}{\sum_{i=1}^{d} \lambda_i} \]

The cumulative EVR tells us what fraction of information is retained when we keep the first $k$ components:

\[ \text{Cumulative EVR}(k) = \sum_{j=1}^{k} \frac{\lambda_j}{\sum_{i=1}^{d} \lambda_i} \]

2.3 Hand-Calculated PCA — Iris Dataset (4 features)

Let us compute PCA on the first 8 Iris samples. Because all 4 features are measured in cm with similar ranges, we only center (we do not divide by std for pedagogical clarity).

Sample	Sepal L	Sepal W	Petal L	Petal W
1	5.1	3.5	1.4	0.2
2	4.9	3.0	1.4	0.2
3	4.7	3.2	1.3	0.2
4	4.6	3.1	1.5	0.2
5	5.0	3.6	1.4	0.2
6	5.4	3.9	1.7	0.4
7	4.6	3.4	1.4	0.3
8	5.0	3.4	1.5	0.2

Covariance Eigendecomposition: $\Sigma = Q\Lambda Q^\top$, producing eigenvalues:

\[ \lambda = [4.228,\; 0.243,\; 0.078,\; 0.024] \quad\Rightarrow\quad \sum \lambda = 4.573 \]

⚠ Variance Preservation Check

Sum of original feature variances = 4.573
Sum of eigenvalues = 4.573
Sum of PC variances = 4.573
PCA preserves total variance — it just re-rotates the axes! ✓

📌 Standardization Rule-of-Thumb

We skipped full standardization here (cm features with similar scales). Always use full standardization when:

Features have different units (age in years vs. income in USD)
Features have vastly different numerical ranges (0–1 percentages vs. 10,000–1,000,000 salaries)
You want equal a-priori importance for every feature

2.4 Case Study — Feature Engineering Showdown (Adult Census Income)

We compare four feature-engineering pipelines on the Adult Income dataset (predict income > $50K / year, ~30K rows, 14 mixed features). Evaluation metric: AUC.

Pipeline	# Features Used	Training AUC	Test AUC	Training Time
Baseline (all features)	14	0.921	0.884	1.0× (ref)
Filter (χ² + ANOVA)	9	0.910	0.881	0.8×
Wrapper (Fwd/Bwd + KNN)	7	0.932	0.876	12.4×
PCA (k = 6 components)	6	0.914	0.888	1.3×

🔍 Case Study Observations (click to expand)

Wrapper had the highest train AUC but lowest test AUC — classic overfitting to the validation folds.
PCA achieved the highest test AUC using only 6 orthogonal components — decorrelation helps generalization.
Filter was fastest, nearly matched baseline, and is a strong first choice for quick iteration.
Wrapper was ~12× slower (re-trains model for every candidate feature set) — expensive at scale.

2.5 Bayes' Theorem — Foundation of Naive Bayes

Reverend Thomas Bayes' 1763 theorem lets us update beliefs given evidence. For a class label $y$ and feature vector $x = (x_1, x_2, \dots, x_d)$:

\[ P(y \mid x) = \frac{P(x \mid y) \cdot P(y)}{P(x)} \]

$P(y)$ — Prior probability of class $y$ (frequency in training data)
$P(x \mid y)$ — Likelihood of features $x$ conditional on class $y$
$P(x)$ — Evidence (same for all classes, so we can ignore it during prediction)
$P(y \mid x)$ — Posterior probability — what we want

2.6 The "Naive" Conditional Independence Assumption

The hard part is $P(x \mid y) = P(x_1, x_2, \dots, x_d \mid y)$ — a full joint distribution over $d$ features is exponentially hard. Naive Bayes makes a strong but computationally convenient assumption:

Naive Assumption: All features are conditionally independent given the class label.

\[ P(x \mid y) = \prod_{i=1}^{d} P(x_i \mid y) \]

This is rarely literally true (features often correlate!), but Naive Bayes works surprisingly well in practice (text classification, spam detection, sentiment analysis) because only the rank order of posteriors matters for classification, not their absolute calibration.

2.7 Naive Bayes Classification Rule

For a new sample $x$, pick the class $\hat{y}$ that maximizes the unnormalized log-posterior (log avoids numerical underflow and turns products into sums):

\[ \hat{y} = \arg\max_{y \in Y} \left[ \log P(y) + \sum_{i=1}^{d} \log P(x_i \mid y) \right] \]

3. Interactive Examples

Example 1: EVR & k-Component Selection

The Wine dataset eigenvalues (sorted) are: $[4.84,\; 2.42,\; 1.55,\; 0.96,\; 0.84,\; 0.66,\; 0.52,\; 0.35,\; 0.31,\; 0.21,\; 0.18,\; 0.15,\; 0.11]$. Click to reveal answers.

(a) What percentage of variance is explained by the first component alone?

Sum of eigenvalues = $4.84 + 2.42 + \dots + 0.11 = 13.0$.
$\text{EVR}_1 = \frac{4.84}{13.0} \approx 37.2\%$.

(b) What is the cumulative variance explained by the first 3 components?

$\frac{4.84+2.42+1.55}{13.0} = \frac{8.81}{13.0} \approx 67.8\%$.

(c) A rule-of-thumb says "keep components until ≥ 95 % variance". Roughly how many components would that be for Wine?

First 9 eigenvalues sum to ≈ 12.4, giving ≈ 95.4 %. So k ≈ 9 components.

Example 2: Bayes' Theorem — Medical Diagnostic

A rare disease affects 1 % of the population ($P(D) = 0.01$). A test is 99 % sensitive ($P(+ \mid D) = 0.99$) and 95 % specific ($P(- \mid \neg D) = 0.95$).

You test positive. What is $P(D \mid +)$? Click to reveal.

Step 1: $P(+ \mid \neg D) = 1 - 0.95 = 0.05$

Step 2: Evidence = $P(+) = P(+\mid D)P(D) + P(+\mid \neg D)P(\neg D)$ = $0.99\!\cdot\!0.01 + 0.05\!\cdot\!0.99 = 0.0594$

Step 3: $P(D \mid +) = \frac{0.99 \cdot 0.01}{0.0594} = \frac{0.0099}{0.0594} \approx \mathbf{16.7\%}$.

Even a 99%/95% accurate test has only ~17% PPV on a 1% prevalence disease (base-rate fallacy!) — always use Bayes.

Example 3: Pipeline Winner Interpretation

Back to the Adult Census case study. Five students interpret the results. Who is correct?

Student	Claim
Ali	Wrapper is best because its training AUC is highest (0.932).
Bilal	PCA is best because its test AUC is highest (0.888).
Chloe	Filter is best because it's fastest.
Dua	Wrapper is overfitting because test AUC (0.876) < train AUC (0.932).

Bilal and Dua are both correct.
Bilal: Test-set performance on unseen data is the gold standard for generalization.
Dua: The large train–test gap for Wrapper is textbook overfitting to validation-fold feedback.
Ali is wrong (train AUC is optimistic), and Chloe confuses speed with quality.

4. Numerical Solutions

Problem 1: PCA Eigenvalues → Components

A 6-feature dataset produces sorted eigenvalues: $\lambda = [3.0,\; 2.0,\; 1.0,\; 0.6,\; 0.3,\; 0.1]$.

📘 Step-by-Step Solution

Step 1: Total variance = $\sum \lambda_i = 3.0 + 2.0 + 1.0 + 0.6 + 0.3 + 0.1 = 7.0$

Step 2: Individual EVRs:

EVR₁ = 3/7 ≈ 42.9 %
EVR₂ = 2/7 ≈ 28.6 % (cum ≈ 71.4 %)
EVR₃ = 1/7 ≈ 14.3 % (cum ≈ 85.7 %)
EVR₄ = 0.6/7 ≈ 8.6 % (cum ≈ 94.3 %)
EVR₅ = 0.3/7 ≈ 4.3 % (cum ≈ 98.6 %)
EVR₆ = 0.1/7 ≈ 1.4 % (cum = 100.0 %)

Step 3: How many PCs to keep for ≥ 90 % cumulative variance?

4 components give 94.3 % ≥ 90 %. So $k = 4$.

Step 4: Compression ratio = $\frac{k}{d} = \frac{4}{6} \approx 67\%$ of the original feature count.

Problem 2: Naive Bayes — Spam vs. Ham

Training corpus: 40 % spam ($P(S)=0.4$), 60 % ham ($P(H)=0.6$). Word frequencies given the class:

Word	$P(\text{word} \mid S)$	$P(\text{word} \mid H)$
win	0.60	0.05
free	0.50	0.10
meeting	0.05	0.50

A new email contains words: {win, free}. Classify it.

📘 Step-by-Step Solution

Step 1: Compute unnormalized log-posterior for SPAM:

\[ \log 0.4 + \log 0.60 + \log 0.50 = -0.916 + (-0.511) + (-0.693) = \mathbf{-2.120} \]

Step 2: Compute unnormalized log-posterior for HAM:

\[ \log 0.6 + \log 0.05 + \log 0.10 = -0.511 + (-2.996) + (-2.303) = \mathbf{-5.810} \]

Step 3: argmax → SPAM (−2.120 > −5.810).

Note: Normalization constant is same for both classes, so we skip it.

Problem 3: Prior × Likelihood Intuition

Now take the exact same vocabulary email {win, free}, but move to a company inbox where only 1 % of mail is spam ($P(S)=0.01,\; P(H)=0.99$).

📘 Step-by-Step Solution

Step 1: log-posterior SPAM = $\log 0.01 + \log 0.60 + \log 0.50 = -4.605 -0.511 -0.693 = \mathbf{-5.809}$

Step 2: log-posterior HAM = $\log 0.99 + \log 0.05 + \log 0.10 = -0.010 -2.996 -2.303 = \mathbf{-5.309}$

Step 3: argmax → HAM (−5.309 > −5.809).

Moral: The same email flips classification because the prior changed. When spam is rare (1%), the evidence of two spammy words isn't strong enough to overcome the low base rate. This is why priors matter!

5. Try It Yourself

Problem 1 — EVR & k Selection

Sorted eigenvalues from a 5-feature PCA run: $[2.5,\; 1.5,\; 0.7,\; 0.2,\; 0.1]$.

Compute the total variance preserved by PCA (hint: sum of eigenvalues).
Compute the individual and cumulative EVR for each component.
How many PCs should you keep for a ≥ 90 % cumulative-variance threshold?

Total variance = 2.5 + 1.5 + 0.7 + 0.2 + 0.1 = 5.0
EVR = [50%, 30%, 14%, 4%, 2%]; cum = [50%, 80%, 94%, 98%, 100%].
Keep 3 components (reaches 94% ≥ 90%).

Problem 2 — Bayes' Theorem in a Factory

Factory machines M1, M2, M3 produce 20 %, 30 %, 50 % of total output respectively. Their defect rates are 5 %, 3 %, 1 %. An item is randomly sampled and found defective.

Which machine is it most likely to have come from? Compute all 3 posteriors.

$P(\text{def}) = 0.2\!\cdot\!0.05 + 0.3\!\cdot\!0.03 + 0.5\!\cdot\!0.01 = 0.01 + 0.009 + 0.005 = 0.024$

$P(M_1 \mid \text{def}) = \frac{0.010}{0.024} \approx 41.7\%$
$P(M_2 \mid \text{def}) = \frac{0.009}{0.024} = 37.5\%$
$P(M_3 \mid \text{def}) = \frac{0.005}{0.024} \approx 20.8\%$

Most likely: Machine M1 (despite producing only 20% of items, its 5% defect rate dominates the posterior).

Problem 3 — Naive Bayes Play-or-Not

Sports dataset: 9 play days, 5 no-play days. Weather frequencies given class:

Outlook	$P(\cdot \mid \text{Play})$	$P(\cdot \mid \text{No})$
Sunny	2/9	3/5
Overcast	4/9	0/5
Rain	3/9	2/5

Classify Outlook = Overcast (apply Naive Bayes; use 1-sample Laplace smoothing where necessary).

Unnormalized posterior (Play) = (9/14) × (4/9) = 36/126 ≈ 0.286

Unnormalized posterior (No) = (5/14) × (0/5) = 0 ← zero! 😱

With Laplace smoothing (α=1) on likelihood:

$P(\text{Overcast}\mid\text{No}) = \frac{0 + 1}{5 + 3} = 1/8$

Smoothed posterior (No) = (5/14) × (1/8) ≈ 0.045

argmax → Play (0.286 > 0.045).

6. Interactive Quiz

Answer all 5 MCQs. Click on an option to get instant feedback.

Your score: 0 / 5

7. Key Takeaways

Explained Variance Ratio: $\text{EVR}_j = \lambda_j / \sum_i \lambda_i$. The scree plot visualizes EVR and the elbow guides $k$-selection (common thresholds: 90%, 95%, or the "elbow").
Variance is conserved under PCA rotation: Sum of eigenvalues = Sum of original feature variances. PCA doesn't "lose" information globally — it reorganizes variance into orthogonal axes.
Standardize before PCA whenever feature scales differ. Without standardization, income (in dollars) will dominate PCA over temperature (in °C).
In the case study, PCA generalized best (highest test AUC), Wrapper overfit (highest train AUC, slowest), Filter was fastest with near-baseline quality. No single method is always best — run the experiment.
Bayes' Theorem: $P(y \mid x) = P(x \mid y)P(y)/P(x)$. The prior $P(y)$ is critical. A "99 % accurate" test on a rare disease still gives a low posterior.
Naive Bayes assumes conditional feature independence: $P(x \mid y) = \prod P(x_i \mid y)$. Take logs to turn products into sums and avoid underflow. Works extremely well on text despite the "naive" assumption.

8. Common Pitfalls

Forgetting to standardize before PCA. The resulting PCs will be meaningless if features are on incomparable scales. Always use StandardScaler for heterogeneous data.
Fitting PCA on the full dataset before train/test splitting. This leaks test-set distribution information. Fit on train only, then apply the learned $W$ to both train and test.
Interpreting individual PCA components as meaningful "features." PCs are linear combinations of all original features and are often not human-interpretable. Use factor analysis if interpretability is critical.
Multiplying probabilities directly in Naive Bayes (not log space). For even moderate $d$, $\prod P(x_i \mid y)$ underflows to zero on floating-point hardware. Always use log-space arithmetic.
Zero-frequency problem (P(word∣class)=0). A single unseen feature zeros the entire posterior. Always use Laplace (add-α) smoothing on categorical Naive Bayes likelihoods.
Confusing "Wrapper overfits on training" with "Wrapper is useless." Wrappers are valid — you just need to couple them with strong regularization, a holdout validation set, and/or use them only on small feature subsets.

9. Resources

scikit-learn: PCA User Guide (with explained_variance_ratio_) — Official docs, Iris & SVM face-recognition example
scikit-learn: Naive Bayes — GaussianNB, MultinomialNB, ComplementNB, BernoulliNB
Machine Learning with PyTorch and Scikit-Learn (Raschka et al.) — Ch. 5 (Dimensionality Reduction) + Ch. 6 (Model Evaluation)
Mitchell, Machine Learning (1997) — Classic Ch. 2: Concept Learning & the Candidate-Elimination Algorithm / Bayes
UCI Adult (Census Income) Dataset — The case-study dataset; reproduce our filter/wrapper/PCA results at home
Kaggle: Naive Bayes for Spam Detection (with TF-IDF) — End-to-end text pipeline
Interactive PCA Visualization (setosa.io) — Drag sliders to see how PCs rotate in 2-D / 3-D

Sample	Sepal L	Sepal W	Petal L	Petal W
1	5.1	3.5	1.4	0.2
2	4.9	3.0	1.4	0.2
3	4.7	3.2	1.3	0.2
4	4.6	3.1	1.5	0.2
5	5.0	3.6	1.4	0.2
6	5.4	3.9	1.7	0.4
7	4.6	3.4	1.4	0.3
8	5.0	3.4	1.5	0.2

Sample	Sepal L	Sepal W	Petal L	Petal W
1	5.1	3.5	1.4	0.2
2	4.9	3.0	1.4	0.2
3	4.7	3.2	1.3	0.2
4	4.6	3.1	1.5	0.2
5	5.0	3.6	1.4	0.2
6	5.4	3.9	1.7	0.4
7	4.6	3.4	1.4	0.3
8	5.0	3.4	1.5	0.2

Sample	Sepal L	Sepal W	Petal L	Petal W
1	5.1	3.5	1.4	0.2
2	4.9	3.0	1.4	0.2
3	4.7	3.2	1.3	0.2
4	4.6	3.1	1.5	0.2
5	5.0	3.6	1.4	0.2
6	5.4	3.9	1.7	0.4
7	4.6	3.4	1.4	0.3
8	5.0	3.4	1.5	0.2